After completing this lesson, you’ll be able to:
Creating self-serve data integration workflows takes the burden of user requests away from expert staff, enabling them to concentrate on more important work. This course explains how to implement self-serve using FME Flow, and how to allow the end-user to set parameters, select format, choose a coordinate system, and define which layers to download.
Self-serve workflows allow the end-user to carry out their own data translations and transformations. Routine data management tasks are offloaded from staff to the user, who is empowered to carry out processes at their own convenience.
Usually, the system is set up in such a way that the end-user needs no prior FME experience or training to carry out their goals; for example, they can access the functionality through a web interface customized to their particular needs. In fact, the user does not even need to know of FME, or that FME is the engine driving their applications!
In general, there are two types of self-serve systems.
Data Upload systems are where the user is able to upload their data to be processed on FME Flow. A typical application would be a user uploading data to be validated. The data is run against a number of tests in an FME workspace and the results are sent back to the user.
Data Download systems are where the user is able to serve themselves with data. A typical application would be an organization that frequently provides data to either staff or customers. With a Data Download system, the user can fetch their own data rather than having to be provided with it in a more manual way. Data can be downloaded as a set of files or streamed directly into an application.
A Data Upload system is one where a user provides data to FME Flow for it to process.
Data Upload can be used from any FME Flow client, including:
Data Upload is often used for submitting data to an organization; for example, a property developer submits a planning application containing a DWG dataset to a municipal planning department.
It is also often used for publishing data to be processed on FME Flow; for example, FME Flow can provide a data validation web service and an end-user would upload a set of data to be checked.
It’s worth noting that data upload also includes not just data, but other resources that may be required for a translation to run; for example, custom transformers or text-file lookup tables may also be uploaded.
A Data Download system is one where a user selects their own choice of data to download.
Data Download can also be used from any FME Flow client, usually:
Data Download operates in a different way than simply running a workspace.
When you run a workspace (with the Job Submitter) the data is written to the location specified by the workspace; for example a file, directory, or database.
Data Download instead writes the output to a zip file and presents the user with a link to that file. This makes it ideal for self-serve because the data is delivered directly to the user.
Self-serve is implemented through a number of Services on FME Flow. A Service is a particular method of communication between client and server. FME Flow provides a wide range of services to carry out different forms of data self-serve.
In the simplest of terms, a service is a piece of software that handles communications between a client and a server. In other words, it’s a tool that allows users to access complex functionality through a simplified interface.
In terms of FME Flow, the client is often—but not always—a web browser that passes requests to FME Flow using a service.
In short, a service allows the sending of specific types of requests to FME Flow and allows results to be provided to client applications in a specific way.
For example, instead of just running a workspace, you can have a web page ask for the results of the workspace as a package of data compressed in a zip file.
A service is just a simpler way of communicating requests to FME Flow than using the REST API. Also, FME Flow includes a number of predefined services that cover a lot of the functionality you are likely to need.
FME Flow includes the following services:
| Data Download Service | Transformation Services |
| Data Streaming Service | |
| Job Submitter Service | |
| KML Network Link Service | |
| Data Upload Service | Utility Services |
| Token Service | |
| REST Service | |
| Notification Service | Notification Services |
Remember that services can communicate in both directions. Transformation services – for example, Data Download – are primarily self-serve tools for Flow to deliver data to the end-user.
Utility services can be described as “helper” services. They interact with FME Flow to assist in menial tasks such as uploading data or providing token security. In most cases, these are facilities that an author or developer will be using in a way that’s hidden from the user.
The Notification Service is used for passing short messages into and out of FME Flow. Incoming messages notify FME Flow to take some action, whereas outgoing messages alert an end-user (or system) that some sort of event has occurred. Automations are powered by the Notification Service.
When a workspace is published to FME Flow the last section of the publishing wizard is for registering it with a particular service:
The Job Submitter service is automatically selected in the FME Flow publishing wizard, whenever a workspace is published, but many other services are available too.
Registering a workspace with a service makes the workspace available for use in that service although, as you’ll discover, not every workspace is capable of being used by every service.
The Edit button to the right of each service allows the properties on how the workspace will use the service to be configured:
Notice how these parameters include ones for notification topics to trigger on completion of the workspace.
A workspace may be registered against one service or many services.
